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the use of ROI with Job Training Partnership Act (JTPA) adult and youth 
training programs identified five benefits: simplicity of interpretation; low 
cost of measurement; usefulness as a local marketing tool; ability to improve 
communications with private industry councils; and ability to best allocate 
resources. Five limitations were also found: lack of control group for 
determining net impact of training services; lack of a consistent methodology 
for comparing returns across local providers; non- JTPA services for which ROI 
models assume credit; nonmonetary benefits that were not accounted for; and 
limited availability of data. Key issues in conducting cross-program ROI 
analysis of Texas' work force development programs at the LWDB level are as 
follows: purpose, perspective, net or gross measurement, within- or 
cross -program, benefit measurement, cost measurement, and timing of 
measurement. Steps have been recommended for implementing an ROI measure. 
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Return on Investment: A Cost-Effectiveness Measure 
for the Texas Workforce Development System 

Since the passage of Texas HB 1863 and SB 642 in 1993, the Texas Council on 
Workforce and Economic Competitiveness (TCWEC) and the agencies involved in Texas’ 
workforce development system have collaborated to develop eight core performance 
measures for the Texas workforce system. Five of these measures have been approved 
and are in the process of implementation, including being incorporated into the 
Automated Adult Learner and Student Follow-Up System maintained by the Texas State 
Occupational Information Coordinating Committee (SOICC). The three remaining 
measures have been approved, but not yet specifically defined. Two additional 
measures— training-relatedness of placement and program cost effectiveness (e.g., return 
on investment) were adopted for further research and consideration. 

This paper presents some of the key issues in developing a retum-on-investment 
(ROI) measure to assess the cost-effectiveness of Texas workforce programs. The main 
focus of the paper will be on issues related to conducting cross-program ROI analysis at 
the level of the Local Workforce Development Board (LWDB) in the environment of 
career/one-stop centers. In the past, most analyses of cost-effectiveness measures, such 
as ROI, have been conducted categorically, by project or by program. But Texas is 
building a new comprehensive, customer driven workforce development system, managed 
at the level of the LWDB to meet the needs of employers, workers, and those preparing 
to enter the workforce. This new systemic approach relies on the coordination of 
multiple funding sources to deliver workforce services to achieve common goals. 
Therefore, the approach to performance management must be systemic as well. 

Benefit-Cost Analysis using Return on Investment 

Benefit-Cost analysis of investments in human capital, such as workforce 
development, can help determine whether these programs should be expanded, contracted, 
or modified to become more cost-effective. One of the fundamental aspects of benefit- 
Cost analysis is determining what benefits and costs result from the provision of services 
and to whom they accrue. The following table catalogs the different types of benefits and 
costs that can accrue to (1) the participants directly benefiting from the human 
investment program, (2) the taxpayers who support the program, and (3) the society as a 
whole (simply the combination of the gains and losses from the participants and 
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taxpayers). This table distinguishes the individual benefits and costs that reflect overall 
social gains and losses from those that reflect internal transfers} 



Table I 

Potential Benefits and Costs from Workforce Development Services 





Participant 


Taxpayers 


Society 


Benefits 


1. earnings gmn 


+ 




+ 


2. increase in taxes paid 




+ 


+ 


3. decrease in transfer payments (e.g., welfare) 


- 


+ 




4. decreased crime and associated costs 




+ 


+ 


5. non-monetary satisfaction 


+ 




+ 


Costs 


1 . expenditure on program 




- 


- 


2. foregone earnings while in training 


- 




- 


3 . foregone taxes while in training 




- 


- 



ROI is a particular method of conducting benefit-cost analysis. It is an appealing 
concept because its meaning seems self-evident and easily understood. However, many 
factors can complicate its calculation or interpretation. Problems with ROI include; 

• the difficulty of accurately measuring an appropriate investment cost figure which 
may call for arbitrary cost allocation judgments or addition of opportunity costs; 

• problems with accurately capturing all benefits resulting from the investment; and 

• the passage of time; investment costs typically come early, while returns may 
come years later; thus, the time value of money (discounting) may need to enter 
the ROI equation; and, it may be difficult to match returns ■mth specific costs. 

Accurate use of ROI as a measure of program or system cost-effectiveness 
assumes that all associated benefits and costs can be quantified and accurately measured. 
To make the most appropriate and effective use of ROI, it must first be determined how 
the measure is to be calculated, including (1) which benefits and costs can and will be 



' Boardman, et al. Cost-Benefit Analysis: Concepts and Practice. Upper Saddle River, New Jersey: 
Prentice Hall. 1996. 
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accounted for, (2) how the costs and benefits are to be measured, (3) from which 
perspective the measure will be calculated, and (4) what time period will be covered. 

The ROI of a project or program is typically measured by comparing the cost of 
the program to the present value of money returned to the economy in increased earnings 
of the participant, increased tax contributions due to employment (e.g., employees 
contribute to the tax base through social security payments, federal and state income 
taxes, and increased sales tax revenues), and decreased welfare costs. Often times, in 
practice, the benefits and costs that are more difficult or costly to measure are excluded 
from the calculation, such as foregone earnings of participants or non-monetary benefits. 
This exclusion can lead to biased measures that provide misleading information about the 
cost-effectiveness of a program. Consequently, an ROI measure should be interpreted in 
light of what benefits and costs were included or excluded from the calculation. 

Also important to ROI analysis is the use of appropriate methods and data 
sources for measuring the relevant benefits and costs of the program. The costs in an ROI 
measure are typically measured in monetary terms by consulting cost accounting 
information within program records. However, for a more comprehensive measure some 
estimation of foregone earnings or foregone tax revenues should also be calculated. The 
benefits in an ROI measure, such as increased earnings, are often more difficult to measure 
in that they accrue over longer periods of time and often require extensive follow-up 
research for accurate measurement; and non-monetary benefits (e.g., increased self-esteem 
of the participant, increased job satisfaction, and the non-monetary value to society of 
having more educated citizens) are the most difficult to identify or measure. Furthermore, 
since human capital investment programs typically make an investment in the present and 
derive a return in the future, all benefits and costs must be calculated in present value 
terms. 

Another important factor in accurate measurement and interpretation of ROI, is 
determining the appropriate perspective from which to calculate the measure. Returns 
from investment in workforce development accrue to the participants, taxpayers, and to 
society as a whole; and ROI can be calculated from any of these perspectives. The 
perspective used will determine what relevant benefits and costs should be measured, and 
how the measure should be interpreted. Most commonly, when ROI is discussed in the 
context of workforce development, the perspective is that of the taxpayer. In that case, 
ROI measures the return to taxpayers per tax dollar spent on workforce development. 
However, an exclusive focus on ROI from the taxpayer perspective may lead to 
underinvestment in workforce development programs if returns accrue mainly to 
participants. In light of this potential problem, focusing on ROI from the society’s 
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perspective would be preferable, since it would potentially lead to a more optimal level of 
investment. ^ 

One cautionary note should be mentioned here with regard to the use of ROI in 
the context of workforce development programs. Most commonly in the business sector, 
ROI is used to refer to the incremental gain from an action (e.g., investment in capital 
equipment), divided by the cost of that action. If an investment costs $100, and returns 
$150 in the same period, it is said to have a 50% ROI. Appropriate methods of either 
cost minimization or benefit maximization can be employed by a firm to maximize their 
ROI for a particular project. However, use of this most simplistic method in the context 
of workforce development programs is highly inappropriate and could lead to program 
decisions with serious unintended consequences (e.g., focusing resources on the easiest- 
to-serve populations, while diverting resources from the more hard-to-serve populations). 
The ROI estimation methodology required in the context of human capital investments is 
somewhat different and more complex than that for capital equipment; and it requires 
different benefit and cost accounting methods and a different relevant time frame. 

Experiences with ROI in JTPA and JOBS 

ROI methodologies have been used in the Job Training Partnership Act (JTPA) 
programs for a number of years in selected local areas in certain states (e.g., Indiana, New 
York).^ In New York, ROI reporting in JTPA requires (among other things) that 
information be gathered on an individual's employment and earnings at program entry and 
then compared to their employment and earning status at some point following his or her 
leaving the program. For JTPA, the follow-up time frame has been thirteen weeks after 
program exit. This approach has been adopted by five Jobs Opportunities and Basic 
Skills (JOBS) program operators in New York state as part of their JOBS/JTPA 
coordination effort. One of these program operators goes further and conducts follow-up 
at thirteen months after entry to employment, in addition to the thirteen weeks after exit 
Two program operators prepare a benefit-cost analysis, similar to an ROI report, which 
is based on earnings gains to participants who enter employment and welfare grant 
savings. 



^ Gramlich, Edward M. A Guide to benefit-cost Analysis. Second Edition. Englewood Cliffs, New 
Jersey: Prentice Hall. 1990. 

^ New York Welfare Reform Task Force. JOBS and JTPA Coordination in New York State. New York 
Department of Human Services: NYATEP Welfare Reform Task Force. May 1995; and Reed, David. The 
Return on Investment from Indiana 's Training Programs Junded through the Job Training Partnership 
Act. Indiana Office of Occupational Development Indian^lis, IN. October 1986. 
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In 1994, a GAO project team studied the concept of ROI for use with JTPA adult 
and youth training programs.^ The team identified five benefits and five limitations of the 
ROI method. The benefits suggested from their study included; 

• simplicity of interpretation 

• low cost of measurement 

• usefulness as a local marketing tool 

• ability to improve communications with private industry councils 

• ability to facilitate the best allocation of resources 



The limitations included; 

• lack of control group for determining the net impact of training services 

• lack of a consistent methodology for comparing returns across local providers 

• non JTPA services for which ROI models assume credit 

• unaccounted for non-monetary benefits such as improved self -esteem 

• and the limited availability of data. 

Key Issues in Texas 

The following section discusses the key issues in conducting cross-program ROI 
analysis of Texas’ workforce development programs at the LWDB level. 

Purpose: Texas must determine the main purpose for the ROI measure. The 
measure could be used as part of a performance-based budgeting strategy (enacted in 
Texas legislation in 1991) to identify and eliminate cost-ineffective programs. Or the 
measure could be used for continuous improvement by identifying low performing/cost 
ineffective LWDB’s so that technical assistance and training can be focused on them. 

Perspective: As mentioned earlier, it is also important to determine the 

appropriate perspective from which to calculate the measure; from the perspective of 
taxpayers, of participants, or of society as a whole. The perspective used will determine 
what relevant costs and benefits should be measured and how the ROI measure should be 
interpreted. Most commonly, when ROI is discussed in the context of workforce 
development the perspective is that of the taxpayer. In that case, ROI would measure the 
monetary return to taxpayers per tax dollar spent on workforce development. But, as 
mentioned earlier, analyzing from the society’s perspective may lead to a more optimal 
level of investment. 



TJ.S. General Accounting Office. Assessment of Return on Investment Analysis for Use in JTPA Adult 
and Youth Training Programs. Washington, D.C.: General Accounting Office. May 1994. 
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Net or Gross Measurement: Another issue is whether the ROI measure will be 
calculated using net or gross input data. This will determine what the interpretation of 
the measure should be. Program benefits (e.g., earnings gains, reduced transfer payments, 
increased tax payments, etc.) can be measured in gross or net terms. Gross measurement 
is straight forward, but net measurement requires benefits to be estimated net of similar 
results for a control or carefully constructed comparison group^. However, Texas has the 
advantage of long-term pre and post-program UI wage records in the SOICC follow-up 
data system, which can help achieve some minimal level of net measurement without the 
use of a control group. This approach would probably be the most effective for use by 
the LWDB’s. 

Within-Program or Cross-Program: In light of Texas’ new systemic approach to 
delivering workforce development services, it must be determined whether the ROI 
should be computed within programs or across programs. Computing ROI across 
programs may increase the number and difficulty of the measurement issues that must be 
resolved, such as incongruent accounting systems, differing program years, etc. However, 
cross-program analysis supports Texas’ systemic approach to delivering workforce 
development services. 

Benefit Measurement. The array of benefits that can be produced from 
investments in workforce development are difficult to determine, and even more difficult 
to measure, including those for the individual (e.g., increases in earnings, improved self- 
esteem, etc.); the government (e.g., increases in tax contributions, decreases in transfer 
payments, etc.); and for the society as a whole (e.g., reduced crime, increased domestic 
spending, etc.). Many of these benefits would be impossible or at least veiy difficult to 
estimate. In addition, the various agencies providing workforce services typically have 
quite different concepts and approaches to capturing the post-program benefits that they 
can measure. These incongruencies will have to be resolved before cross-program analysis 
can be conducted. Furthermore, since part of the purpose for career/one-stop centers is 
to allow individuals to secure their own workforce services, questions arise concerning 
just how such benefits will be fully captured. Scxne of the most important benefits 
expected to come from this service delivery system is the reduction in the time it takes for 
individuals to access and secure services, a benefit which has often been among the least 
effectively measured. 



’ Bell, Stephen, et. al. Program Applicants as a Comparison Group in Evaluating Training Programs: 
Theory and a Test. W.E. Upjohn Institute for Employment Research; Kalamazoo, Michigan. 1995. 
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Cost Measurement: Accounting for the entire cost of the investment expenditure 
is essential to the accuracy of the ROI measure. However, the major workforce 
development programs in Texas each have different methods of accounting for costs, 
different units for measurement or data collection, and different accounting periods. 
These incongruencies will, no doubt, present serious barriers to conducting cross program 
ROI analysis. In addition, in most programs, there is currently no method for associating 
program costs with related services at the individual level. These cost issues will have to 
be.resolved before accurate data can be collected to support ROI measurement. 

Timing of measurement: How often will the measure be computed? It is unlikely 
that ROI measures can be accurately computed very frequently, since it may require 
years to see the complete results of service outcomes. Even though ROI is a legitimate 
expectation of a public program, such returns are expected to accrue over a relatively Iwig 
period of time; and in public efforts to upgrade human capital, it may not be reasonable to 
expect the period of return to coincide with the period of investment. 

Next Steps 

It seems that the most effective use of ROI in the Texas workforce development 
system would be a net impact ROI measure (i.e., controlling for pre and post UI earmngs 
levels) that is calculated across programs at the LWDB level, and used for continuous 
improvement of the LWDB’s service delivery. It is at the LWDB level where the 
multiple funding streams (state/federal) will be directed to participants and the service 
delivery decisions will take place with maximum flexibility to respond to performance 
management information; and it is also at that level that the cost accounting is the most 
comprehensive and the most conducive for the purposes of calculating ROI. In addition, 
regardless of the specific formula used to calculate ROI, the measure should be interpreted 
and presented in light of the specific benefits and costs that were included in the 
calculation, and not assumed to have included benefits or costs that were not measurable. 

Given the issues discussed above there a number of steps that the state of Texas 
can take to proceed toward implementing an ROI measure for Texas workforce 
development programs. These include the following, among others: 

1 . Review the cost accounting practices of the major workforce development programs 
in Texas to better understand their important differences. Questions must be resolved 
on the issues of definition of terms, units of measurement, accounting methods, level 
of aggregation, data collection and reporting of program costs. 
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2. Review the concepts and approaches to capturing post-program benefits used by the 
major woricforce development programs in Texas to better understand the important 
differences and to develop consensus around appropriate concepts. Consistent 
measurement of benefits is essential for valid ROI analysis. 

3. Determine a consistent methodology for estimating ROI at each LWDB, and 
determine which benefits and costs can reasonably be included in the calculations. 

4. Choose an appropriate LWDB as a pilot site to test and refine the cross-program 
ROI measure before full statewide implementation. 
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